Bayesian Ordinal Aggregation of Peer Assessments: A Case Study on KDD 2015
نویسندگان
چکیده
Peer assessment is the most common approach to evaluating scientific work, and it is also gaining popularity for scaling evaluation of student work in large and distributed classes. The key idea is that each peer reviewer or grader rates a relatively small subset of the items, and that some method of manual, semiautomatic, or fully-automatic aggregation of all assessments defines the eventual rating of all items – the grade in peer grading, or whether to accept or reject a scientific manuscript. In this paper, we explore in how far a Bayesian Ordinal Peer Assessment (BOPA) method can provide additional decision support when making acceptance/rejection decisions for a scientific conference. Using data from the 2015 ACM Conference on Knowledge Discovery and Data Mining (KDD), where this system was deployed, we discuss the potential merit of the BOPA approach compared to conventional decision support offered by the Microsoft Conference Management System (CMT).
منابع مشابه
Comparison of Maximum Likelihood Estimation and Bayesian with Generalized Gibbs Sampling for Ordinal Regression Analysis of Ovarian Hyperstimulation Syndrome
Background and Objectives: Analysis of ordinal data outcomes could lead to bias estimates and large variance in sparse one. The objective of this study is to compare parameter estimates of an ordinal regression model under maximum likelihood and Bayesian framework with generalized Gibbs sampling. The models were used to analyze ovarian hyperstimulation syndrome data. Methods: This study use...
متن کاملGGRA: a grouped gossip-based reputation aggregation algorithm
An important issue in P2P networks is the existence of malicious nodes that decreases the performance of such networks. Reputation system in which nodes are ranked based on their behavior, is one of the proposed solutions to detect and isolate malicious (low ranked) nodes. Gossip Trust is an interesting previously proposed algorithm for reputation aggregation in P2P networks based on t...
متن کاملProbabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs
With the enormous scale of massive open online courses (MOOCs), peer grading is vital for addressing the assessment challenge for open-ended assignments or exams while at the same time providing students with an effective learning experience through involvement in the grading process. Most existing MOOC platforms use simple schemes for aggregating peer grades, e.g., taking the median or mean. T...
متن کاملA factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments
Evaluating open-response assignments in Massive Open Online Courses is a difficult task because of the huge number of students involved. Peer grading is an effective method to address this problem. There are two basic approaches in the literature: cardinal and ordinal. The first case uses grades assigned by student-graders to a set of assignments of other colleagues. In the ordinal approach, th...
متن کاملSelf-, Peer-, and Teacher-Assessments in Writing Improvement: A Study of Complexity, Accuracy, and Fluency
Alternative assessment approaches and, among them, self-assessment and peer-assessment are becoming increasingly important in educational contexts. Designed to compare self-assessment, peer-assessment, and teacher-assessment, this study included 90 EFL students from 3 intact classes divided into 3 groups: self-assessment, peer-assessment, and teacher-assessment. After taking the TOEFL Proficien...
متن کامل